Tidal Ventures

In today's fast-paced technology landscape, managing risks associated with the use of Generative AI (Gen AI) is increasingly important for software companies of all sizes.

The potential risks of Gen AI applications are a constant topic of discussion in the global tech industry. This has led to a pressing need for companies to understand and manage these risks effectively. The importance of aligning the risks assessed with each company's unique business objectives has been emphasized in numerous conversations with industry leaders.

If you hold a leadership role in technology or security, and software is crucial to your company's operational success, it's vital to understand the risks associated with adopting Gen AI. It's equally important to prioritize these risks and allocate the necessary resources to address them.

Key questions to consider include:

• What are the risks associated with using Gen AI, such as issues related to sensitive data masking, access control, and restrictions on using third-party Gen AI tools and services?

• What are the risks involved in integrating Gen AI-powered features into your company's products and services?

As security practitioners, we often focus primarily on the first set of risks, especially the use of Gen AI products by employees. However, leaders need to make informed decisions about where to invest their resources, particularly since there are often existing methods to manage these risks. The second set of risks, related to the integration of Gen AI into the company's offerings, may present less straightforward solutions and will largely depend on how Gen AI is incorporated into the company's products. This is where security leaders can add significant value.

"Risk Management Should Not Only Focus On How Employees Use Gen Ai But Should Also Consider How The Company's Business Could Be Negatively Affected By Managing The Second Set Of Gen Ai-Associated Risks"

When assessing the use of Gen AI within their companies, security leaders should consider potential risk areas such as:

• Hallucinations

• Toxicity

• Sensitive Data (PII/PHI/PCI/IP)

• Prompt Injections

• Model Poisoning

This list is a starting point and is not exhaustive. The integration of Gen AI into a company's applications, products, and services could potentially lead to negative business impacts. Leaders need to identify which risks align with their company's current challenges since these risks could significantly impact the company's operations, revenue, and customer relationships.

Risk management should not only focus on how employees use Gen AI but should also consider how the company's business could be negatively affected by managing the second set of Gen AI-associated risks.

Security leaders accustomed to working with DevOps, Site Reliability Engineering, Infrastructure, IT, and related teams should consider expanding their collaboration when assessing potential risks from Gen AI. Consider speaking to other teams, including:

• Machine Learning (AI/ML infrastructure, pipeline, etc.)

• Data Engineering (Datalake, ETL, other data-related infrastructure)

• Data Science (Model development, testing, usage)

• Product Engineering (Feature, functionality development, customer-facing)

Lastly, numerous motivated teams, projects, and researchers are currently working on solving many of these problems. Here’s a non-exhaustive list of resources:

• Nightfall AI - Sensitive Data

• Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

• Mithril Security - Sensitive Data

• Aimon-Rely - Model Quality Continuous Monitoring

• Lakera - LLM Security

• Humanloop - LLM Collaboration and Evaluation

• Credal - LLM Governance

Risk management is crucial in the rapidly evolving domain of Generative AI (Gen AI). Tech and security leaders need to understand and prioritize these risks, particularly in the context of Gen AI usage and product integration. These risks encompass areas such as hallucinations, toxicity, and sensitive data. Effective risk assessment can be achieved through collaboration with teams beyond DevOps, IT, and Infrastructure, including Machine Learning, Data Engineering, Data Science, and Product Engineering.